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EXECUTIVE SUMMARY 

Naval Aviation is a hazardous undertaking, but in spite of its inherent risk, its 
Class A Flight Mishap (FM) rate has been cut in half for each decade from 1950 to 1990. 
Over the last decade, however, the proportion of aircraft losses in which human error has 
been cited as a contnbutor has remained relatively constant. To address human factors 
issues in flight mishaps, the Human Factors Quality Management Board (HFQMB) was 
established in 1996. By using Mishap Data Analysis (MDA), Organizational 
Benchmarking (OB), and Command Safety Assessment (CSA), the efforts of the 
HFQMB resulted in a significant reduction in FM incidence from the perspective of 
aircrew operations. 

Although human error in maintenance is a smaller contributor, it has been shown 
to be a factor in nearly one in five Class A FMs. To address human error in maintenance, 
the Human Factors Analysis and Classification System - Maintenance Extension 
(HFACS-ME) was developed to classify error types in maintenance. Since Naval 
Aviation is shown to be a high reliability organization (HRO) defined as an organization 
that operates in hazardous environment with less than its fair share of accidents, it shares 
common characteristics with other HROs. These common characteristics are outlined in 
the Model of Organizational Safety Effectiveness (MOSE) and are process auditing, 
quality, reward system, risk management and command and control. Military aviation has 
communication/functional relationships as a sixth component. The augmented MOSE 1s 
the basis of the Maintenance Climate Assessment Survey (MCAS), which is used to 


evaluate the organizational safety climate from the perspective of the maintenance 


XI 


personnel. The MCAS consists of six demographic items and 43 perception items. Each 
of the 43 perception items maps into a single component of the augmented MOSE. These 
questions are collapsed into six component scores for each respondent. 

Although MCAS has been shown to be an effective tool for evaluating the safety 
environment in a maintenance organization, demographic factors and their potential 
relationship with maintainer responses have not been investigated. This study evaluates 
MCAS responses from 894 maintenance personnel of the 3 MAW, and looks at how the 
demographic factors of maintenance personnel might be biasing the component scores of 
the MCAS. The results of this thesis are intended to further refine MCAS demographic 
factors and provide Squadron Commanders with insight into the construct of their 
maintenance personnel. 

The component scores are fitted using the demographics as explanatory factors. 
Univariate analysis is performed for each component using simple models without 
interaction and also with models using two-factor interactions. These models are then 
simplified in order to reduce the number of terms to a more manageable level. 

The results of this thesis show that up to two-factor interaction, the demographic 
factors of the MCAS poorly account for the variance in the responses. The reliance on 
Subjective perception in the scoring 1s the cause of the large amount of variance. Since 
variance cannot be explained by the demographic factors, the MCAS appears to be 
demographically unbiased. Input from subject matter experts is used to refine the 
demographic factors. These revised factors are more usable for Squadron Commanders in 


that they provide more insight into the make up of the maintenance organization. 
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I. INTRODUCTION 

A. OVERVIEW 

Naval Aviation 1s a hazardous undertaking, but in spite of its inherent risk, its 
Class A Flight Mishap (FM) rate has been cut in half for each decade from 1950 and 
1990 (Naval Safety Center, 1997). Class A Mishaps are defined as Naval aircraft 
incidents resulting in death, permanent disability, or property loss or damage in excess of 
one million dollars (OPNAV 3750.6Q, 1989). A flight mishap (FM) is defined as those 
mishaps in which there is $10,000 or greater DoD aircraft damage or loss of a DoD 
aircraft, and intent for flight for DoD aircraft existed at the time of the mishap. Other 
property damage, injury, or death may or may not have occurred. Naval Aviation 
consistently maintains high levels of operability coupled with less than its fair share of 
accidents (Goodrum, 1999). Naval Aviation also possesses the requisite characteristics of 
a high reliability organization (HRO): process auditing, reward system, quality, risk 
management, and command and control. For these reasons, Roberts (1988) labeled Naval 
Aviation an HRO. 

Although Naval Aviation is successful in reducing its Class A FM rate, over the 
last decade the proportion of aircraft losses in which human error has been cited as a 
contributor has remained relatively constant at four of five FMs (Naval Safety Center. 
2000). In 1996, a Human Factors Quality Management Board (HFQMB) is established 
after 17 Class A FMs occurred in only 75 days, climaxing when a Navy F-14 crashes into 
a Nashville, TN neighborhood, to address human factors issues related to mishaps 
(Nutwell & Sherman, 1997). The goal of the HFQMB is to cut the current Class A FM 


rate due to human error in half by year 2000 (HFQMB Charter, 1996). The HFQMB 


adopts three approaches to identify sd target factors contributing to human error: 1) 
Mishap Data Analysis (MDA), 2) Organizational Benchmarking (OB), and 3) Command 
Safety Assessment (CSA). 

MDA establishes the development of HFACS, which is used to identify and 
prioritize human factors contributors to FMs. Among others, it determines inadequate 
supervision and aircrew violations are significant contributors (Shappel & Wiegman, 
1997). Using OB which explores programs which influence aircrew performance, the 
HFQMB determines use of feedback mechanisms in commercial airlines improve crew 
resource management training benefits (Nutwell & Sherman, 1997). Finally, a CSA 
survey, based on a model of HROs, is developed to determine a command’s safety 
posture from an aircrew perspective. This survey finds that 55% of the Navy respondents 
and 65% of the Marine Corps respondents feel that their commands are committed 
beyond what available resources can provide (Ciavarelli & Figlock, 1997). These 
combined efforts make significant progress toward the HFQMB’s goal as evidenced by 
fiscal year 1999 being the safest year in Naval Aviation history in terms of Class A FM 
rate. 

Maintenance is shown to be a contributing factor in nearly one in five Class A 
FMs (Naval Safety Center, 2000). Additionally, during FY90-97, Class C FMs account 
for 75% of all maintenance related mishaps (MRMs). Maintenance is one area where 
hazards can be controlled and risk can be managed while an aircraft is on the ground. 
Much work is done in the field of human factors in maintenance safety for commercial 
airlines (“Human Factors in,” 2000). In 1988, the Aviation Safety Research Act (ASRA) 


mandates close study of aging aircraft structures and human factors affecting safety 


(“History,” 2000). In the spirit of this mandated study, Boeing finds that incomplete 
installation (34%), damaged on installation (15%), improper installation (11%) and 
equipment not installed/missing (11%) were the top contributors in maintenance error 
(Komarniski, 2000). This investigation and classification of types of human error in 
maintenance leads to the development of Boeing’s Maintenance Error Decision Aid 
(MEDA), a system that aids operators and maintainers in the investigation and mitigation 
of maintenance related errors (Allen, Rankin, & Sargent, 1998). 

The ASRA 1s one of the precursors for the FAA’s current goal to reduce the fatal 
accident rate 80% by 2007 as compared to 1994-1996 baseline data (FAA, 1998). Several 
key initiatives are the stepping stones for this ultimate goal: 1) the development of a 
maintenance resource management system; 2) establishment of new _ training 
requirements; 3) implementation of technical advances in aircraft maintenance at repair 
stations; 4) enforcement of safety recommendations from the National Transportation 
Safety Board (NTSB); and 5) recommendations for aging systems maintenance. Little 
work until recently is done for military aviation, and recent efforts involve studying 
forms of maintenance error (Schmorrow, 1998) and the perceived maintenance safety 
climate (Baker, 1998). From a proactive perspective, efforts must be made to continue 
developing assessment tools to identify potential areas for risk management and control 
of conditions before a mishap occurs. 

Using the Maintenance Climate Assessment Survey (MCAS), Goodrum (1999) 
and Oneto (1999) are able to show the prototype survey effectively evaluates a 
maintainer’s perception of safety in maintenance operations. They also further refine the 


MCAS into a present 43 question format. One aspect of the MCAS yet to be addressed 1s 


the demographic categories to determine their potential relationship with maintainer 
responses. Given the structure of maintenance organizations within aircraft communities 
vary, it is unlikely that any pair of samples will have equal proportions of all 
demographic variables. It is anticipated that individual demographic characteristics may 
influence MCAS responses and are therefore potentially biasing the results. By 
understanding the effects of the demographics, one can understand if they impact their 
organization’s safety climate. 
B. BACKGROUND 

The 3 Marine Air Wing (MAW) is a combat-ready expeditionary aviation force 
capable of short-notice worldwide employment to Marine Air Ground Task Force 
(MAGTF) fleet and unified commanders. It is composed of 28 squadrons divided into 
four Marine Air Groups (MAGs) based in Southern California and Arizona. Each MAG 
has 1ts own combat mission: MAG 11 provides air support to MAGTF commanders; 
MAG-13 provides close-air support, conducts armed reconnaissance, and assumes 
limited air-defense roles; MAG-16 transports and resupplies Marine air and ground units; 
and MAG-39 provides utility helicopter support, close-in fire support, fire support 
coordination, aerial reconnaissance, observation and forward air control in aerial and 
ground escort operations during ship-to-shore movement and subsequent operations 
ashore. The aircraft used in these missions are AH-1Ws, UH-1Ns, CH-53s, CH-46Es, 
F/A-18Ds, F/A-18s, AV-8s and C-130Ts. 

From 1990 to 1996, maintenance is a causal factor in 17 percent of all Naval 
Aviation class A FMs (Naval Safety Center, 1997). From April 1997 to July 1999, 


maintenance, maintenance personnel or maintenance depot 1s cited as a causal factor in 


14 FMs (eight class C FMs, four class B FMs, and two class A FMs) experienced by gr 
MAW. The Commander of 3 MAW requests the assistance of the School of Aviation 
Safety (SAS) at the Naval Postgraduate School, which in turn provides safety and risk 
management training to personnel, mishap data analysis, and administration of safety 
climate surveys to help locate problems in the organization. 

C. PROBLEM STATEMENT 

Human error in aviation is an issue that needs to be addressed, and it is recognized 
that the organization has an impact on factors that lead to it. Organizations that possess 
the attributes of a HRO tend to generate environments conducive to the reduction or 
control of human error and consequently experience fewer mishaps. Organizations 
aspiring towards the reduction of mishaps need to assess their safety posture as it relates 
to the attributes of HROs. The School of Aviation Safety at the Naval Postgraduate 
School has developed surveys to assess HRO characteristics in the operational 
environment for aircrew and maintenance personnel. 

The 3 MAW in an attempt to improve its safety posture enlists to have the 
School of Aviation Safety employ the MCAS survey to assess maintainer perception of 
HRO characteristics in its recent operations. These results are revealing, however in order 
to provide for better interpretation of the results, an exploration of the demographic 
variables is in order. This will help commanders to target more effectively specific areas 
of the organization that require attention. 

The current version of the MCAS is administered to the 3 MAW during the last 


half of 1999. Using statistical methods, the collected data is analyzed to assess 


differences in responses that are correlated to differences in demographics. This thesis 
explores the following questions: 
1. Are there measurable demographic effects to the responses on the MCAS? 


2. Is there enough information in the demographics to be used in an adjustment 
process of the overall scores? 


3. Can the MCAS be refined further to either collapse or expand demographic 
factors? 
D. SCOPE AND LIMITATIONS 
Active duty U.S. Marine Corps Squadrons maintenance personnel of the a 
MAW are surveyed during the fall of 1999. Only those squadrons with a representative 
number of respondents are used in the survey. Chapter II provides a basis for 
understanding human error, organizational safety culture, high reliability organizations 
and the assessment of a safety climate. Chapter III presents a discussion of the 
methodology used in this study. Results of data analysis are presented in Chapter IV. 
Chapter V summarizes previous chapters and provides conclusions and recommendations 
as they relate to the material. 
E. DEFINITIONS 
This thesis uses the following definitions (DON, 1989): 
Naval Aircraft. Refers to U.S. Navy, Naval Reserve, U.S. Marine Corps, and U.S. 
Marine Corps Reserve aircraft. 
Mishap. A Naval Aviation mishap is an unforeseen or unplanned event that directly 
involves naval aircraft, which result in $10,000 or greater cumulative damage to naval 


aircraft or personnel. The mishap is further divided into three classes based on the 


amount of damage to the aircraft, property and personnel injury. The following are 

the definitions of the three classes: 

a. Class A. A mishap in which the total cost of property damage (including all 
aircraft damage) is $1,000,000 or greater; or a naval aircraft is destroyed or 
missing; or any fatality or permanent total disability of a person occurs with direct 
involvement of naval aircraft. 

b. Class B. A mishap in which the total cost of property damage (including all 
aircraft damage) is $200,000 or more but less than $1,000,000 and/or a permanent 
partial disability, and or the hospitalization of five or more personnel. 

c. Class C. A mishap invented the total cost of property damage (including al] 
aircraft damage) is $10,000 or more but less than $200,000 and/or injury results in 
one or more lost workdays. 

Mishap rate. The total number of Class A,B and C mishaps per 100,000 flight hours. 
MCAS. A 43-question survey used to gain insight into the maintenance community’s 
perception concerning aviation mishaps within the Navy and Marine Corps. 
HFACS-ME. A taxonomic system used to classify causal factors that contribute to 
maintenance related mishaps. 
HRO. High-Reliability Organization, is an organization that operates in a hazardous 
environment, yet produces very low rate of accidents and incidents, operating effectively 
and safely and having the characteristics of leadership, sound management policies, 
SSE TE standardization, adequacy of resources and staffing, a defined system for risk 


management, and other factors. 
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II. LITERATURE REVIEW 

A. HUMAN ERROR 

Reason (1990) defines error as a planned sequence that fails to achieve its 
intended outcome in the absence of external influence. He (1997) later describes error 
types as active or latent. Where the effects of active errors are often immediate and 
confined, latent conditions lie dormant until set off by a chain of local events and can be 
contributing factors in a variety of failures. This model of latent conditions and active 
failures is adopted by the Naval Safety Center to investigate Class A FMs with respect to 
aircrew error, and was the basis for the development of Human Factors Analysis and 


Classification System (HFACS) (Shappel & Wiegman, 1997). 
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Table 1. Likelihood of Performance Problems in Given Activities. 








Reason (1997) also models human error in the scope of universal human activities 
(see Table 1) and the likelihood of performance problems within each of these types of 
activities. With HFACS, the Naval Safety Center is able to address human error in 
normal control and emergency control conditions. But Reason asserts that maintenance is 
the area with the highest likelihood of human error because maintenance related activities 
are almost exclusively reliant on human performance in the three areas of hands on, 


criticality and frequency. Even with progress in technology, human fallibility remains 


constant (Reason, 1997) and with the frequency of planned maintenance compounded 
with the many pairs of fallible human hands working on exceptionally complicated 
systems, Reason’s model ts a chilling prediction of 3 MAW’s situation. 

To address the maintenance related activity as a contributor to the total of human 
errors in FMs, Schmidt, Schmorrow and Hardee (1998) extend HFACS to specifically 
address the maintenance component of accident causation with the HFACS-Maintenance 
Extension (ME). The HFACS-ME expands upon Reason’s model of latent states and | 
psychological precursors to unsafe acts. Reason (1990; 1997) differentiates these latent 
States in that they create the potential for human error. HEACS-ME classifies latent states 
in the maintenance environment, with three levels of error causation under four 
categories of conditions. The first order conditions are broad precursor categories 
(supervisory conditions, maintainer conditions, working conditions and maintainer acts) 
that are further divided into more specific precursors to human error of the second and 
third order (see Table 2). 

The causes of human error are many. Wickens, Gordon and Lui (1997) state that 
human error can be induced by “inattentiveness, poor work habits, lack of training, poor 
decision making, personality traits, social pressures, and so forth” (p. 427-428). Redmill 
& Rajan (1997) note that a common component in accidents is a worker’s loss of 
concentration which can be caused by “boredom, disinterest, distraction, or attempt to do 
two or more things at once” (p.12). Wickens, Gordon and Lui (1997) assert that the most 
common type of maintenance related error is that of omission. Considering that the nature 
of planned maintenance is to frequently disassemble, inspect then reassemble 


components, it is clear to see the high probability of human error in maintenance (Reason 
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1997). But as Perrow and Reason point out, the errors themselves are seldom isolated, but 


often a single event in a causal chain (Perrow, 1984), or encouraged, or at least not 


prohibited from occurring by latent conditions in the organization (Reason, 1997). 


First Order 


Supervisory Conditions 


Second Order 


Unforeseen 


Squadron 


Maintainer Conditions Medical 
Crew Coordination 


Readiness 


Environment 


Working Conditions 


Equipment 
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Maintainer Acts Error 


Violation 





Third Order 


Hazardous Operations 
Inadequate Documentation 
Inadequate Design 
Inadequate Supervision 
Inappropriate Operations 
Failed to Correct Problem 
Supervisory Violation 
Mental State 

Physical State 
Physical/Mental Limitation 
Communication 
Assertiveness 
Adaptability/Flexibility 
Preparation/Training 
Qualification/Certification 
Violation 

Lighting/Light 
Exposure/Weather 
Environmental Hazards 
Damaged 

Unavailable 
Dated/Uncertified 
Confining 

Obstructed 

Inaccessible 

Attention 

Memory 
Rule/Knowledge 

Skill 

Routine 

Infraction 

Exceptional 


Table 2. HEFACS-ME Levels of Error Causation. 
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B. ORGANIZATIONAL SAFETY CULTURE 

1. Definition 

Organizational culture is defined as shared values and beliefs that interact with an 
organization’s structures and control systems to produce behavioral norms (Uttal, 1983). 
All organizations have their own engineered culture whether good or bad. A safety 
culture is ideal for complex organizations and 1s defined as the product of individual and 
group values, attitude, competencies, and patterns of behavior that determine the 
commitment to, and the style and proficiency of, an organization’s health and safety 
programmes (Booth, 1993). 

2. Composition 

Redmill and Rajan (1997) state that there are three general aspects of safety 
culture: awareness, commitment and competence. Awareness must be present in all 
aspects of design, management and decision making. When a safety mishap does occur, 
it is Commitment that drives the leadership of the organization to investigate and locate 
contributing factors and take immediate action to prevent another occurrence. 
Competence is a combination of education, training, professionalism and personality 
traits that are appropriate for a given task or job (Redmill & Rajan, 1997). 

Reason (1997) prefers the term informed culture, and he divides informed culture 
into four subcultures: reporting culture, just culture, flexible culture and learning culture. 
Reporting culture is “an organizational climate in which people are prepared to report 
their errors and near-misses.”’ Just culture is an “atmosphere of trust in which people are 
encouraged, even awarded, for provided essential safety-related information — but in 
which they are also clear about where the line must be drawn between acceptable and 
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unacceptable behavior.” Flexible culture involves “shifting from the conventional 
hierarchical mode to a flatter professional structure, where control passes to task experts 
on the spot, and then reverts back to the traditional bureaucratic mode once the 
emergency has passed. Such adaptability is an essential feature of the crisis-prepared 
organization.” Learning culture is “the willingness and the competence to draw the right 
conclusions from its safety information system, and the will to implement major reforms 
when their need is indicated.” 

Safety culture has powerful effects. First, it is self perpetuating where workers 
learn from each other and encourage each other to work in a manner consistent with the 
Organization’s safety culture. People are quick to follow the example of coworkers, even 
if this means a lack of vigilance in safety (Redmill & Rajan, 1997). Wogalter, Allison, & 
McKenna (1989) assert that “people are extremely susceptible to social norms; they are 
likely to engage in safe or unsafe behaviors to the extent that others around them do so.” 
Safety culture is also self preserving as new workers learn to identify acceptable methods 
of accomplishing work and are able to pass those standards along to new employees, 
good or bad (Redmill & Rajan, 1997). 

C. HIGH RELIABILITY ORGANIZATIONS 

1. Definition 

Roberts (1990) and Libuser (1994) explain that High-Reliability Organizations 
(HROs), organizations that operate in a hazardous environment, yet produce very low 
rates of accidents and incidents, which operate effectively and safely have certain key 
characteristics in common. Reason (1997) calls them organizations with less than their 


fair share of accidents, and “highly complex, technology-intensive organizations that 
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must operate, as far as humanly possible, to a failure-free standard.” He also explains 
that HROs manage complex technologies that have very low tolerance for error, yet 
maintain the flexibility to successfully function in environments of extreme intensity. 

2. Characteristics of HROs 

Examples of HROs are the nuclear power industry, petrochemical industry, and 
airline industry. Additionally, Figlock (1998) identifies Naval aviation is an HRO. 
Although diverse in purpose, Roberts and Libuser believe these organizations share 
several common characteristics: leadership style, management policies, procedures 
Standardization, superior training, a reward system that recognizes safety achievement, 
adequacy of resources and ane effective management of risks associated with 
hazardous operations, and other factors. 

HROs have a requisite variety. As Weick (1987) states, having diverse people 
from diverse backgrounds and experiences builds requisite variety that is required for 
relatively simple humans to operate complex systems. Additionally, this diversity is 
essential in problem solving, as individuals will approach the same problem uniquely, so 
that the collective contribution is greater than any one individual’s input. 

HROs typically exhibit a high degree of training. Weick (1987) notes that 
“training for the operation of high reliability systems is often tough and demanding so 
that the faint of heart and the incompetent are weeded out.” This is because HROs are not 
afforded the luxury of trial and error. Training is often in the form of simulation and 
Stories. Stories have a big affect on the reliability of an HRO by lending experience to the 


inexperienced: 


The basic idea is that a system which values stories, story tellers, and 
storytelling will be more reliable than a system that derogates these 
substitutes for trial and error. A system that values stories and storytelling 

is potentially more reliable because people know more about their system, 

know more of the potential errors that might occur, and they are more 

confident that they can handle those errors that do occur because they 

know that other people have already handled similar errors (Weick, 1987, 

pelo), 

By sharing the experiences of skilled personnel, novices do not have to learn from their 
own mistakes and are also granted the insight of the skilled. 

HROs appear bureaucratic and uneventful on the surface. There is a strict chain of © 
command in place that dictates policy, procedure and environment. This strong 
centralization 1s apparent during periods of relatively low intensity, but as intensity of 
operating increases, the true nature of the HRO is revealed, where flexibility, delegation, 
improvisation and technical expertise dominate (Reason, 1997; Weick, 1987). This is 
how HROs can be simultaneously centralized and decentralized. Responsibility and 
judgment remain centralized while creativity, improvisation and unsupervised problem 
solving become decentralized in environments of high intensity (Weick, 1987). 

Reliability in HROs is another deceptive aspect. Weick (1987) calls reliability a 
“dynamic non-event,” meaning a great deal of effort goes into ensuring nothing happens. 
This dynamicism is based on the belief that reliability is fleeting and systems tend to 
move to states of unreliability unless constantly maintained: 

Part of the mindset for reliability requires chronic suspicion that small 

deviations may enlarge, a sensitivity that may be encouraged by a more 

dynamic view of reliability (Weick, 1987, p. 119). 


It is the organizational culture for personnel to look for problems before they happen. 


Weick continues that because of the invisibility of the dynamics behind reliability, there 


is a perception that reliability is easily achievable and is only noticed in the presence of a 
breakdown. 
D. ASSESSING SAFETY CLIMATE 

1. Safety Space 

Reason (1997) states that organizations can be mapped into a safety space which 
is acontinuum of degrees of susceptibility to accidents (see Figure 1). Organizations with 
higher resistance will generally have fewer mishaps while organizations with higher 
vulnerability will generally experience more mishaps. No organization is immune. 
Chance, unforeseen circumstances, failures in defenses and human error can cause even 
the most resistant organizations to experience accidents. Within the safety space, currents 
tend to push organizations away from the extremes of resistance or vulnerability and 
toward the center, a compromise between the two. If an organization has the desire to 


become more resistant, it must swim upstream. 


Increasing Resistance Increasing vulnerability 





Organizations 


Figure 1. Organizational Safety Space. 


The effort required to move in the direction of increased resistance must be put 
into reactive and proactive measures. Reason (1997) contends that investigating mishaps 
to find causal factors to be addressed is not even half the battle. To effectively move the 
organization, mishap investigation must be coupled with the identification of conditions 
needing correction, and regular checks. 

2. MOSE and MCAS 

Libuser’s (1994) current Model of Organizational Safety Effectiveness (MOSE) is 
based on work by Roberts and is a categorization of the common characteristics of 
HROs. These characteristics are mapped into five components: 1) Process Auditing (PA) 
- checks by members to identify hazards; 2) Reward System (RS)- expected rewards or 
disciplinary action used to shape behavior; 3) Quality Assurance (QA)- promotion of 
quality performance; 4) Risk Management (RM)- system to identify hazards and control 
operational risks; and 5) Command and Control (CC)- safety climate, leadership 
effectiveness /policies, and procedures for mitigating risks. These components are very 


similar to the aspects of Reason’s (1997) informed culture (Table 3). 


Libuser’s MOSE Components Reason’s Informed Culture 






Reward System (RS) Just Culture 









Quality Control (QA) 


Risk Management (RM) Flexible Culture 
Command and Control (CC) Flexible Culture 


Table 3. Comparison of Libuser’s MOSE and Reason’s Informed Culture. 


Reporting Culture 






Ciavarelli and Figlock (1997) adapt the MOSE for use in Naval Aviation using 
practices and terminology of that environment and develop the Command Safety 
Assessment, a survey that addresses each of the MOSE categories from the viewpoint of 
the aircrewman. This survey is administered to 1,254 aviators revealing that 
organizational and supervisory issues are seen by aircrewmen as impacting flight safety. 

The Maintenance Climate Assessment Survey (MCAS) is the product of the 
implementation of the MOSE and CSA in a maintenance context. Baker (1998) starts by 
reducing 155 candidate questions to 67 items that specifically addressed aviation 
maintenance. Augmenting Libuser’s (1994) five category MOSE model with a sixth 
category, Communication/Functional Relationships, Baker (1998) modifies the CSA to 
look at aviation safety from the point of view of the maintenance person. Using 
regression techniques, Baker is able to further reduce the survey into a compact 35 item 
form, with almost all questions mapped to a single category of the augmented MOSE. 

Goodrum (1999) and Oneto (1999) show that the MCAS is a valid tool to 
accurately assess an aviation maintenance environment, but note that some items in the 
survey need restructuring. Oneto notes that these items address more than one category of 
the MOSE. Their inputs help change the MCAS to its current 43 item format (see 
Appendix A). While Goodrum and Oneto are able to show content validity in the MCAS, 
there is much left to examine. Since there is no known or accepted measure for MCAS 
results and providing feedback to the concerned squadrons, an effort is underway to 
explore concurrent validity in the survey on a per question basis (Schmidt, personal 


communication). Questions that have a low response mean are noted as areas that need 


attention, corresponding to a particular category of the MOSE, and which part of the 
HRO needs closer examination. 

Additionally, by looking at squadron mean scores to the survey and available 
mishap data, Harris (personal communication) is looking at the MCAS predictive validity 
in the incidence of mishaps within a squadron based on adjusted mean scores. While the 
work by Harris seeks to broaden the scope and applicability of the MCAS, this study 
explores internal aspects of the survey. By separating responses by demographics, this 
thesis will further explore the attitude of the Naval aviation maintenance person with 
respect to safety. 

3. Instrument Design and Demographics 

The design of the MCAS is a cross-sectional one time look at a maintenance 
Organization (see Appendix A). It is a self-administered questionnaire that polls 
respondent perception about the safety of their working environment at all levels. 
Demographic items preserve anonymity by excluding personal questions which could be 
linked to individuals allowing personnel the freedom to express their true opinion to the 
perception items (Oneto, 1999). 

Though there are many methods for data collection from people, surveys, when 
designed properly, are very effective for recording respondent scores based on a 
particular model (Fink & Kosecoff, 1985). MCAS responses are forced using a five point 
Likert scale (see Appendix A). Using a forced scale instead of subjective comments 
“ills for rapid compilation of data and analyzation of responses as numerical values, 
which is convenient for numerical scoring and comparisons of scores between 


respondents, or groups of respondents (Fink & Kosecoff, 1985). The grouping is 
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facilitated by the six demographic items of the survey (Baker, 1998). By being able to 
group types of maintainers independent of their scores, it 1s possible to investigate for 
unequal perceptions across demographic groups. 

The demographic items in the MCAS capture aspects of the maintainer within the 
Organization such as experience, and rank, and ignore personal information such as age, 
race, sex and ethnicity (Baker, 1998). Although it would be difficult to capture every 
possible combination of demographic factors, the MCAS 1s able to capture most job types 
from most aircraft communities. Each of the demographic line items in the MCAS 
represents a simple factor that might influence the scoring. It is unknown which of these 
demographics constitutes a valid or invalid factor. 

E. SUMMARY 

HROs are complex and dynamic by nature but are not impervious to unsafe trends 
in human performance. Perrow (1984) points out that regardless of an organization’s 
structure and nature, “normal accidents” will continue to occur. With HROs, these 
accidents tend to occur less frequently, but the consequences of the accidents tend to be 
large i magnitude. These organizations must put effort into swimming upstream through 
Reason’s safety space towards increased resistance to accidents. This happens through 
reactive measures like mishap analysis and proactive measures to identify “pathogenic 
conditions” (Reason, 1997). Though reactive measures are in place, proactive measures 
are coming up to speed. 

Two critical parallel developments in organization safety theory in regards to 
aviation safety are the development of effective taxonomies (e.g., HFACS-ME) and the 


identification and accurate modeling of HROs (e.g., augmented MOSE). The 
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development of CSA and MCAS are steps to link the taxonomies and the organization 
models. Since MOSE parallels Reason’s informed culture, it is possible with MCAS to 
identify those conditions that are not conducive to safety and take proactive measures to 
move the organization towards increased resistance to accidents in the safety space. Since 
3° MAW is a HRO, CSA and MCAS allow for the identification of the MOSE 
components that require attention. 

The MCAS has been revised into a more usable form, and has been validated, 
showing that the individual items do address specific MOSE components. This thesis is 
another step into revising the MCAS further by looking at the validity of the demographic 
factors. Mapping the demographic factors into the MOSE component scores will show if 
the factors are relevant or can be removed from the survey. Additionally, if these factors 
do not account for the variance in scores, the survey is not asking the correct 


demographic questions and will require further revision. 
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I. METHODOLOGY 

A. RESEARCH APPROACH 

The intent of this study is to assess the maintainer’s perception of safety in his or 
her work environment. This research involves the use of analysis techniques to partition 
the collected data into smaller groups based on demographics then investigate differences 
in responses among the groups. If statistical differences are found, a comparison between 
the group mean and a particular squadron mean shows how a particular demographic 
differs from the rest of the squadron. Conversely, it shows which demographics have 
response means that are more reflective of the squadron means. 
B. DATA COLLECTION 

1. Subjects 

Surveys are administered to 977 officers and enlisted personnel responsible for 
Naval Aviation maintenance. These subjects come from squadrons and maintenance units 
of the 3 Marine Air Wing located at MCAS Miramar, CA, Camp Pendleton, CA, and 
MCAS Yuma, AZ. The aircraft represented are the AH-1 “Super Cobra,” UH-1 “Huey,” 
CH-53 “Super Stallion,’ CH-46E “Sea Knight,” F/A-18D “Night Attack Hornet,” F/A-18 
“Hornet,” AV-8B “Harrier,” and the C130T “Hercules.” 

Additionally, subject matter experts are interviewed about what they consider to 
be important demographic information about the personnel in their maintenance 
organizations. Subject matter experts are military aviators with at least eight years active 


duty service. The results of these interviews are given in the next section. 
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2. Instrument 

The MCAS is a self-administered, group survey consisting of two parts: 1) 
demographics: 2) perception. Part I captures demographic factors of each subject: 
community, squadron, rank, years of aviation maintenance experience, work center, and 
shift. There are eight choices available for community, with an additional option of 
“other.” The squadron factor records the three-digit squadron designator. Embedded in 
these two factors is aircraft type (seventh factor). Rank is divided into four levels of 
enlisted personnel and three levels of officer personnel. Years of aviation maintenance 
experience 1s partitioned into seven levels. Work center or shop, is divided into eight 
shops with the option of “‘other.”’ Shift divides subjects into dayshift or nightshift. 

Part II captures subject perception of his or her work environment. There are 42 
items, each of which is mapped into a single augmented MOSE component: process 
auditing (six questions), reward system (eight questions), quality assurance (S1x 
questions), risk management (nine questions), command and control (eight questions), 
and communication/functional relations (six questions). Each perception item asks 
subjects to rank a specific safety related activity or aspect of their organization using a 
five point Likert rating scale with verbal anchors as follows: Strongly Disagree, Disagree, 
Neutral, Agree, Strongly Agree. When completed, the items for each MOSE component 
are averaged to attain six composite scores, each one corresponding to the subject’s rating 
of that particular augmented MOSE component for his or her organization. 

3. Procedure 

The survey is administered on site and in a group setting at the various 
participating Squadrons of the 3 MAW. Additionally, the survey is given in 
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conjunction with a scheduled maintenance safety presentation on human factors issues in 
aviation. The Squadrons are in various stages of training and operational tasking at the 
time of the survey being administered. The variety of operational tasking with which the 
squadrons are simultaneously involved during the administration of the MCAS accounts 
for much of the variance in the number of surveys collected from each squadron. 
Potential MCAS respondents are briefed on maintenance issues, the survey and its 
purpose and questions that arise pertaining to the survey are answered by the survey 
administer. Respondents fill out the surveys using scannable computer forms. The 
surveys are then immediately collected upon completion to allow for maximum 
accountability. 

For personal interviews, subject matter experts report what they think are the 
important factors when considering the demographics of personnel in their maintenance 
organizations. Notes are taken during these interviews and a list is compiled from the 
responses of the personnel interviewed. This list represents potential MCAS demographic 
items. 

C. DATA ANALYSIS 

1. Data Tabulation 

Survey results were compiled into a database using a scanning machine, then 
imported into Microsoft Excel. The spreadsheet consists of rows of respondents and 
columns of survey items (both demographic and survey items). Demographic items 
eco mainly bivariate and multivariate responses, such as squadron (aircraft type 
embedded), rank, and years experience. Survey item responses were assigned a numerical 


value of 1 through 5 corresponding to the Likert scale, with higher values being assigned 
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to more positive responses (strongly agree) and lower values assigned to more negative 
responses (strongly disagree). Each of the questions in the survey addresses one of the 
six MOSE components. Items addressing similar MOSE components were collapsed into 
an average score for that particular component. Any items that were missing values were 
excluded and not averaged into the component score. No weighting is assigned to items 
in the event of a missing item score. The three demographic response items 
corresponding to the respondent’s squadron number were collapsed into a single coded 
value. 

2. Statistical Analysis 

Microsoft Excel is used to provide summary Statistics and initial familiarization 
with the data. The data were cleansed by removing subject responses with omitted 
demographic items leaving 894 responses. Items corresponding to augmented MOSE 
components were averaged for each respondent, leaving six scores along with 
demographic response items. If subjects omitted a perspective item response, the 
component score is averaged for the completed items of that component. No weighting 
for omitted perspective items is administered. Histograms of the demographic make up of 
the data are constructed. The data are then exported to ARC for its powerful graphing 
capabilities. Initial scatterplot matrices of each component over all demographic factors 
revealed no linear, exponential, or power trend, although scatterplots matrices of the 
components over themselves revealed a linear trend. 

The data are then transported to MathSoft S-plus for analysis. Categorical 
demographics are coded as factors and the aov function is implemented to fit the six 


component scores based on demographic factors. Six models (one for each component) 
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are constructed for a regression without interaction among factors. Six additional models 
are fit for a regression looking at two factor interaction. Each model is then simplified by 
using S-plus to remove unimportant terms. Models are compared and similar models for 
some of the components are found. Three term interaction models are not explored in this 


thesis. 
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IV. RESULTS 
A. SIMPLE MODELS WITHOUT INTERACTION 
A scatterplot matrix is constructed to see if there are any trends in the data. This is 
done without designating the independent variables as factors. The scatterplot shows that 
there is some relationship between the component scores, but little information about the 
demographics is revealed. Figure 3 is a scatterplot matrix for PA versus each of the 
demographic factors. Visual inspection reveals a possible relationship between PA and 


Rank and Total years of Aviation experience. 











Figure 1. Scatterplot of PA versus Demographic Factors. 


Three dimensional bar plots for the components versus each factor are constructed 
to see if there are any visual clues as to some type of relationship between the factors and 


the response. No linear, log linear or exponential relationship is visible. However, the 
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plots seem compatible with those of normally distributed histograms for each level of 
factor, all centered in the same approximate region of the component score (e.g. Figure 


4). 





Figure 2. Three Dimensional Bar Plot of PA Scores. PA scores 
are plotted against different levels of Total Years of Aviation 
Maintenance Experience. 


Since there is no indication that a transformation of the data is required due to 
visible trends, linear models for the six individual components are fit against 
untransformed factors without interaction using the aov function in S-plus. Model 
checking plots are constructed and case 219 is shown to have very high influence in all 
models. Case 219 is an E-6/7 with 15-20 years of aviation maintenance experience, 
works the day shift in “other” work center, in a VMH squadron. Although there is no 
significance test associated with Cook’s Distance, case 219 is deleted from the data set 
due to its unusually high influence and new models were fit (see Appendix C). 

Model checking plots are constructed to check the fit of the models with case 219 


removed. The scatter plot of the data with the regression superimposed reveals what 
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might be a slight upward trend in the response for all components (see Appendix D). The 
histograms of the residuals show that the distribution of the residuals appear to be normal 
(see Appendix E), and the scatterplot of the residuals versus the fitted values shows no 
discernable pattern in the residuals (see Appendix F). The QQ-plot shows that the 
residuals are thin at both tails for all models (see Appendix G), but the normal shape is 


tenable. 
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CC 0.1280 0.6349 


0.1233 0.6846 


Table 4. R-squared and 6 for Models without Interaction. 





Values for the coefficient of determination, R*, show that these models account 
but poorly for the variance in the data. The best model is RM, accounting for less than 
19% of the total variance (see Table 4). The model with the lowest R* is PA with only 
11% of the total variance explained. Values for standard error, G, indicate that there is a 
large spread in the response values. For example, with a perfect R’ of 1, the model for PA 


tells us that 68% of the respondents score between 3.31 and 4.42, and that 96% score 


a1 


between 2.76 and 4.97. Since the only possible scores are between | and 5, relatively 
high values for 6 are not much help with understanding the data. 

The results from the analysis of variance from each model causes rejection of the 
null hypothesis that all of the coefficients in the mode] are zero, accepting the alternate 
hypothesis that at least one coefficient is not equal to zero (see Appendix H). 
Additionally, the models for PA, CC and CR fail the lack of fit test indicating that the 
shape of the fit is not correct. These models do not do well in describing the data. 

B. MODELS WITH TWO FACTOR INTERACTION 

Two-factor interaction models are constructed for all of the components using S- 
plus to see if more of the variance in the data can be modeled and model checking plots 
are constructed. The scatterplot of the data with the superimposed regression shows a 
linear trend in the response against the factors (see Appendix J), and the distribution of 
the residuals appears to be normal (see Appendix J). The scatterplot of the residuals 
versus the fitted values shows no discernable pattern in the residuals (see Appendix K). 
The QQ-plots show strange behavior at values close to zero, but that they are close to 
being normal for all components (see Appendix L). R° and & are given in Table 5. 

These two term interaction models are better at explaining more of the variance in 
the data as indicated by the values for the coefficients of determination, however there is 
very little reduction in the values for the standard error. This improvement in the values 
for R° comes at the cost of increased complexity in the models. While the models without 
interaction have 43 terms, the two term interaction models have 343 terms. By adding 
300 terms to the model, 300 degrees of freedom are lost resulting in no significant 


improvement in the standard error. 
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0.4968 0.5996 


0.4850 0.6523 


Table 5. R’ and 6 for Two Factor Interaction Models. 





Analysis of variance on the two factor interaction models causes rejection of the 
null hypothesis that all of the coefficients are equal to zero. The alternative hypothesis 1s 
accepted meaning that at least one coefficient is not equal to zero. Present in the anova 
tables is evidence that some of the terms are not necessary in the model and that 
simplification is possible. 

C. REDUCED MODELS WITHOUT INTERACTION 

The step function in S-plus is used to subtract terms from the simple models in an 
effort to simplify the models without losing too much of the information they provide. S- 
plus accomplishes this by using Akaike’s information criterion which 1s of the form: 

AIC ==JlO cea hie on i en) ee 
where r is the total number of estimated parameters. The AIC is a value that penalizes a 


model for having high complexity when compared to simpler models with fewer terms. 
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The step function is applied to the simple models without interaction and the results are 
given in Appendix N. The reduced models have between 15 and 33 terms compared to 43 
terms for the original models. Model checking plots are constructed with no significant 
graphical differences between the reduced simple models and the simple models (see 
Appendices O-R). Values for the coefficient of determination and standard error are 
given in Table 6. As expected, less of the total variance is explained by the reduced 
models, and the value for 6 increases. 
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0.0915 0.5532 


OA 0.1002 0.6345 


0.1240 0.5925 


0.1071 0.6365 


0.0784 0.6906 


Table 6. R* and 6G for Reduced Models without Interaction. 





D. REDUCED TWO FACTOR INTERACTION MODELS 

The step function in S-plus is applied to the two factor interaction models and the 
results are given in Appendix S. The data were fit to the new models and model checking 
plots are constructed and given in Appendices T-W. The scatterplot of the data with the 
Superimposed regression line indicates that there might be a linear relationship between 
the factors and the score, but most of the data looks like a point cloud. The histograms of 
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the residuals have the appearance of a normal distribution. The predicted values versus 
the residuals have no pattern and the QQ-plots look normal. 

The reduced models have significantly decreased values for R° with little change 
in values for 6. These models use between 33 and 59 terms which is a significant 
simplification over the 343 term models, but a lot of the explanation of the total variance 


is lost in the transition. Table 6 summarizes R’ and 6 for the reduced models. 


Component 


ars 0.1098 0.5496 
— 0.1228 0.6287 
0.1854 0.5802 


—— 0.2453 0.5534 
omens 0.1747 0.6225 
0.1342 0.6779 


Table 7. R’ and & for Reduced Two Factor Models with Interaction. 





E. COMPARING MODELS 

The S-plus anova function is applied to pairs of models to see if they are 
Statistically different. The results are given in Appendix U. At a = 0.05, eight pairs of 
models were found to be statistically similar, with the highest similarity between simple 
and reduced two term interaction models for PA and QA (p-Value > 0.99). Most of the 


model pairs are statistically different. 
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Choosing the most appropriate model for a complex data set is compromise 
between a model that explains enough of the data while being simple enough to use. The 
two factor interaction models are too complicated to be practical with 343 terms. The rest 
of the models are not different enough to distinguish in practice, so the models with the 
fewest terms are the preferred models, which in this case are the reduced simple models. 
Realistically, none of the models are useful. Not enough of the total variance in the data 
is explained nor is the range of expected scores reduced to a useful level. 

F. INTERVIEWS 

Subject matter experts interviewed agree that while all of the factors in the survey 
are important, additional factors could be added to more effectively group personnel and 
provide more information about the maintenance organization. They also think that 
dividing total years of aviation maintenance experience into two items, years worked in 
MOS and years worked outside of MOS 1s necessary to clarify that Total years of 
aviation maintenance experience is not simply time on active duty. The experts also 
indicate factors that look at levels of education and training and levels of morale and 
motivation are important and should be included in the demographic items of the survey. 


Responses from subject matter experts are given in Appendix Y. 
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V. CONCLUSIONS 
A. FINDINGS 

The results of this thesis show that at the first level of interaction, the 
demographic factors of the MCAS poorly account for the variance in the responses. The 
models constructed using linear regression and analysis of variance do not capture the 
responses of the surveyed population, showing that the demographic factors have low 
utility in data analysis. While analysis of variance shows that the models are preferred to 
no mode] at all, in use the models are too complex and do not provide enough insight into 
the surveyed group. 

Since the component scores are subjective perceptions, there is no correct score to 
any of the perception items. The reliance on the human component in the scoring 1s the 
cause of the large amount of variance, and since variance cannot be explained by the 
demographic factors, the MCAS appears to be demographically unbiased. The three 
dimensional bar plots of the component scores versus the levels of factors seem to 
support this (see Figure 4). Either the MCAS has insignificant biasing across factors or 
the present factors do not correctly group respondents to allow the biasing to be 
conclusively measured. 

Although the demographics do not effectively group respondents, they do provide 
information about the demographic composition of the surveyed group. This information 
alone can be useful to commanders in understanding the substance of their squadrons. To 
sel these items more useful, changes to the MCAS demographic items based on the 


responses from subject matter experts are recommended below. 
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B. RECOMMENDATIONS 

With the MCAS moving to the internet, it will be possible to use more 
demographic factors than what is currently constrained by the layout of scannable 
computer response sheets. The MCAS demographic items should be changed to include 
the following items: 

1) Check the box corresponding to your community: 

2) Type in your unit number. 

3) Type in the number of months have you been with your current squadron. 


4) Type in the total number of maintenance activities to which you have been 
assigned. 


5) Type in the number of deployments you have made. 

6) Check the box corresponding to your rank. 

7) Type in the number years have you worked in your MOS. 

8) Type in the number of years have you worked outside of your MOS. 
9) Type in any supervisory designations that you hold. 

10) Check the box corresponding to your work center. 

11) Check the box corresponding to your shift. 


12) Have you attended A School? 


With these new questions, further analysis can be conducted to investigate for 
valid factors that properly describe the data in the responses, in addition to investigating 
if personal performance makes a difference in scoring. Additionally, MCAS could be 


modified to gauge safety climates in other military activities such as military ordnance 
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handling facilities and flight deck operations by adjusting the demographic items to suit 


those specific activities. 
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APPENDIX A. 43-ITEM MAINTENANCE CLIMATE ASSESSMENT SURVEY 
MAINTENANCE CLIMATE ASSESSMENT SURVEY (MCAS) 


Purpose: The MCAS was designed to capture maintainer perceptions of maintenance operations as they 
relate to safety. Your responses help guide Naval Aviation’s on-going efforts to reduce aviation related 


mishaps. Thank you in advance for your participation! 

Directions: Do not write on this form. Fill in all of your responses using the computer sheet provided. 
Fill in each box that corresponds to your response completely using a pencil. This is not a timed event, so 
answer each question carefully and honestly. Individual responses will not be reported, only compiled 
results will be provided to each squadron. 


Part I- Demographics has six items requesting unit and biographical data. This information will aid in the 
response analysis. NO attempts will be made to identify individuals. 


Part IJ- Perceptions has 43 questions pertaining to the maintenance operations. Please choose the response 
to each item that most correctly reflects your honest opinion. Responses are: 


A- Strongly Agree B- Agree C-Neutral D-Disagree E- Strongly Disagree 


Part I- Demographics 


Line | Fill in the numbered circle corresponding to your community? 
(1) VMGR (2) VMA (3) VMFA (4) HMT (5) HMM 
(6) VMAQ (7) HMH (8) VMH (9) Other 

Line 2-4 Fill in the circles corresponding to your squadron number 

Line 5 Fill in the numbered circle corresponding with you rank 


Cieles (2) E4-5 (3) E6-7 (4) E8-9 (5)WOl1-4 (6) O1-03 (7) O4-5 


Line 6 Fill in the numbered circle corresponding to your total years 
of Aviation Maintenance experience 
(1) <1 (2) 1-2 (3)3-5 (4)6-10 (5) 11-15 (6) 15-20 (7) 20+ 


Line 7 Fill in the numbered circle corresponding to your work center 
(1) Power Plants (2) Airframes (3) Survival (4) Quality Assurance 
(5) Ordnance (6) Avionics (7) MAINT Control (8) Line (9) Other 


Line 8 Fill in the numbered circle corresponding to your shift 
(1) Day (2) Night 


Part II Perceptions 
Fill in the lettered circle that corresponds with your response to each item. 


SA A N D SD 

1. The command adequately reviews and updates safety. (A) (B) (C) (D) (E) 

SA A N D SD 

2. The command monitors maintainer qualifications and has (A) (B) (e} (D) (E) 
a program that targets training deficiencies. 

SA A N D SD 

3. The command uses safety and medical staff to identify/ (A) (B) (C) (D) (E) 


4] 


manage personnel at risk. 


4. CDIs/QARs routinely monitor maintenance evolutions. 


5. Tool control is taken seriously in the command and 
support equipment licensing 1s closely monitored. 


6. Signing off personnel qualifications are taken seriously. 


7. Our command climate promotes safe maintenance. 


8. Supervisors discourage SOP, NAMP, or other procedural 
violations and encourage reporting safety concerns. 


9. Peer influence discourages SOP, NAMP, or other 
violations and individuals feel free to report them. 


10. 


Violations of SOP, NAMP, or other procedures are not 
common in this command. 


. The command recognizes individual safety achievement 


through rewards and incentives. 


. Personnel are comfortable approaching supervisors 


about personal problems/illness 


Safety NCO, QAR, and CDI, are sought after billets. 


. Unprofessional behavior is not tolerated in the command 


. The command has a reputation for quality maintenance 


and sets standards to maintain quality control. 


QA and Safety are well respected, and are seen as 
essential to mission accomplishment. 


. QARs/CDIs sign-off after required actions are complete 


and are not pressured by supervisors to sign-off. 


. Maintenance on detachments 1s of the same quality 


as that at home station. 


. Required publications/tools/equipment are available, 


currenvUServiceable, and used. 


. QARs are helpful, and QA 1s not “feared” in my unit. 


. Multiple job assignments and collateral duties adversely 


affect maintenance. 
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sD 


(E) 


SD 


(E) 


SD 
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sD 


(E) 


sD 
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SD 


(E) 


SD 


(E) 


SD 
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SD 


(E) 


SD 


(E) 


SD 


(E) 


SD 


(E) 


SD 


(E) 


SD 


(E) 


SD 


(E) 


SD 


(E) 


Z2. 


25. 


24. 


20: 


Z0) 


Lt 


28. 


29) 


30. 


ail, 


32; 


a3) 


34, 


25: 


36. 


BE 


38. 


: 


Safety is part of maintenance planning, and additional 
training/support is provided as needed. 


Supervisors recognize unsafe conditions and manage 
hazards associated with maintenance and the flight-line. 


I am provided adequate resources, time, personnel to 
accomplish my job. 


Personnel turnover does not negatively impact the 
command’s ability to operate safely. 


Supervisors are more concerned with safe maintenance 


than the flight schedule, and do no permit cutting corners. 


Day/Night Check have equal workloads, and staffing is 
sufficient on each shift. 


Supervisors shield personnel from outside pressures and 
are aware of individual workload. 


Based upon my command’s current assets/manning it 
is not over-committed. 


My command temporarily restricts maintainers who are 
having a problems. 


Safety decisions are made at the proper levels and 
work center supervisor decisions are respected. 


Supervisors Communicate command safety goals and 
are actively engaged in the safety program. 


Supervisors set the example for following to 
maintenance standards and ensure compliance. 


In my command safety is a key part of all maintenance 


operations, and all are responsible/accountable for safety. 


Safety education and training are comprehensive and 
effective. 


All maintenance evolutions are properly briefed, 
supervised, and staffed by qualified personnel. 


Maintenance Control is effective in managing all 
maintenance activities. 


Good communication exists up/down the chain of 
command. 


I get all the information I need to do my job safely. 
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(E) 


SD 
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SD 
(E) 


SD 
(E) 


sD 
(E) 


40. 


41. 


43. 


Work center supervisors coordinate their actions. 


My command has effective pass-down between shifts. 


flight. 


Maintainers are briefed on potential hazards associated 


with maintenance activities. 


. Maintenance Control troubleshoots/resolves gripes before 
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SA 


(A) 


SA 


(A) 
SA 
(A) 


SA 


(A) 


SD 


(E) 


SD 
(E) 
SD 
(E) 


sD 
(E) 


APPENDIX B. MODEL OF SAFETY EFFECTIVENESS COMPONENTS. 


COMPONENT 1: PROCESS AUDITING 


— 


sad ok ae 


6. 


The command adequately reviews and updates safety practices. 

The command monitors maintainer qualifications and has a program that targets 
training deficiencies. 

The command uses safety and medical staff to identify/manage personnel at risk. 
CDIs/QARs routinely monitor maintenance evolutions. 

Tool Control is taken seriously in the command and support equipment licensing 1s 
closely monitored. 

Signing personal qualifications are taken seriously. 


COMPONENT 2: Reward System and Safety Climate 


— 
° 


7 


Our command climate promotes safe maintenance and flight operations. 
Supervisors discourage SOP, NAMP or other procedure violations and encourage 
reporting safety concern. 

Peer influence discourages SOP, NAMP or other violations and individuals feel free 
to report them. 

Violations of SOP, NAMP or other procedures are not common in this command. 
The command recognizes individual safety achievement through rewards and 
incentives. 

Personnel are comfortable approaching supervisors about personal problems/illness. 
Safety NCO, QAR, and CDI, are sought after billets. 

Unprofessional behavior is not tolerated in the command. 


COMPONENT 3: QUALITY ASSURANCE 


Il. 


Zz 


A 


The command has a reputation for quality maintenance and has set standards to 
maintain quality control. 

QA and Safety are well respected, and are seen as essential to mission 
accomplishment. 

QARs/CDIs sign-off after required actions are complete and are not pressured by 
supervisors to sign-off. 

Maintenance on detachments is the same quality as that at home station. 

Required publications/tools/equipment are available, current/serviceable, and used. 
QARs are helpful, and QA is not “feared” in my unit. 
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COMPONENT 4: RISK MANAGEMENT 


2: 


Multiple job assignments and collateral duties adversely affect maintenance. 

Safety is part of maintenance planning, and additional training/support is provided as 
needed. 

Supervisors recognize unSafe conditions and manage hazards associated with 
maintenance and the flight line. 

I am provided adequate resources, time, personnel to accomplish my job. 

Personnel turnover does not negatively impact the command’s ability to operate 
safely. 

Supervisors are more concerned with safe maintenance than the flight schedule, and 
do not permit cutting comers. 

Day/Night Check have equal workloads, and staffing is sufficient on each shift. 
Supervisors shield personnel from outside pressures and are aware of individual 
workload. 

Based upon my command’s current assets/manning it 1s not over-committed. 


COMPONENT 5: COMMAND AND CONTROL 


— 
. 


My command temporarily restricts maintainers who are having problems. 

Safety decisions are made at the proper levels, work center supervisors decisions are 
respected. 

Supervisors communicate command safety goals and are actively engaged in the 
Safety program. 

Supervisors set the example for following to maintenance standards and ensure 
compliance. 

In my command Safety is a key part of all maintenance operations and all are 
responsible/accountable for safety. 

Safety education and training are comprehensive and effective. 

All maintenance evolutions are properly briefed, supervised, and staffed by qualified 
personnel. 

Maintenance control is effective in managing all maintenance activities. 


COMPONENT 6: COMMUNICATION / FUNCTIONAL RELATIONSHIPS 


RWN > 


WA 


Good communication exists up/down the chain of command. 

I get all the information I need to do my job safely. 

Work center supervisors coordinate their actions. 

My command has effective pass-down between shifts. 

Maintenance Control troubleshoots/resolves gripes before flight. 

Maintainers are briefed on potential hazards associated with maintenance activities. 
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APPENDIX C. INFLUENCE PLOTS FOR AUGMENTED MOSE 
COMPONENTS 
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APPENDIX D. SIMPLE MODELS: SCATTERPLOTS WITH SUPERIMPOSED 
REGRESSION LINE 
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APPENDIX E. SIMPLE MODELS: HISTOGRAMS OF RESIDUALS 
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: RESIDUALS VERSUS FITTED 


APPENDIX F. SIMPLE MODELS 
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APPENDIX G. SIMPLE MODELS: QQ-PLOTS 
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APPENDIX H. SIMPLE MODELS: ANALYSIS OF VARIANCE 
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APPENDIX I. TWO FACTOR INTERACTION MODELS: SCATTERPLOTS 
WITH SUPERIMPOSED REGRESSION LINE 


PA QA 


foes) o°0 wo wr oom Oo comT Mm 0 oO OC 
@ooo mam foo oo ° owo ° 
° o @@°oaxceD oo ie] oo @ @raumpo apo 


CO COOGNCDEIDEEDO | aD See we coo 


oo oO 60890 COC 0 ° 
oOo © CO @OUwEE CONEDDO «6©«@O @Oo CUneEmODoo «9 ClO 


° FD COTA AARRINEEEETI 7 o OS GmEMEEaoEDS 0° 
© 0 C@OTEEEEEEXD «© 
COO CRRA 

©  COGKEEDeEEED axnuOcomED © 

@ OlmDenxD cane co 

0 COORD 0 ConmEDO capa GND 


° O@® GRIESE ©@o 


Et Eee 
Surs ae 8 t mm cc 


9° 


io] OO GocIniemOD CED COO £0 
O OGDCERRUEKITED COD GO 
ZOD 0 CCKEEEXOCEEIO ID COO 


PA 
QA 
3 


oo DM @ © coOo@e Do °o 
OD omc MO 00 00 
o @® a co oo 90 
oo 6 ooo 6 o o 
o oO 0 ° 
° ° N ° ° 0° 
° 


CQOOMGDOe GEOG OO @M oO 
©0O00OGDaED oO a 000 2 
' ©0000 moo @ o ° 
o @ axe 2000 

nr) 
° 


° 





2 3 4 5 1 2 3 4 $s 


Fitted : (lactor(Community) + factor(Squadron) + factor{Rank) + tactor(Tot. Years) + factor(Shop) + factor(Shii Fitted : (factor(Community) + factor(Squadran) + factor{Rank) + factor(Tot. Years) + factor(Shop) + factor(Shr 


RS RM 


o Oo@ 090 o 9 
oo 
oo@mo oo 
co oma oo 00 
ap@m o om 000 
oO9 GO Wan OW 0O 


RS 


© 9@CamaQNXD COOMO Oo 
© O@@@D Wo Oo WO Oo 
@o 0 OHO amo ° 
o M00 0 apo oo 0 
° © © 600 00 ° 
oaD °° Ye) ° 
° ° ° ° 
°° 





2 3 4 Ss 1 2 3 “ 


Fitted - (tactor(Community) + factor(Squadron) + factor(Rank) + factor(Tot. Years) + factor(Shop) + factor(Shi Fitted « (factor(Community) + factor(Squadron) + factor(Rank) + factor(Tot. Years) + factor(Shop) + factor(Shs 


CC CR 













° ° w oO qoam oo o 0 
rs OED oo fa 2] Oo D0 foo] 
g oo o°0 ° ODO MME 00 0 COUlC CO 
° ° 2oOo amo © © @m Oo 0000 
° oa omomo - = Boe oe) ee 
So enema arin cnn ° 0 Cpe? atoomocap °° 
°o OC @® CEOS Giz cr oo wa oo @ i i rr Cia oo 
° ao eee o> SG © © OCD gEEDEETOD 0 co 
° ° * = “ 2 ° OO Ctra @ COO © 
O@@0 © coos CODED O © OGMND CCOMMMEEEEED 00 0 0 
oe oe © O@ om 
Oe OEE 
ED O O GEO MD C=O ° ~ foe) © Oo 
8 Exp ance ao O aKDO o.° COOOREEDO COG COEEED GcO 
ape 0 Dam Oo @ oO ao @o 
oo Oaboo Commo 
0,0 @o apo ° © ommpo oa@nm~ 00 am o 0 
200 oaqp Oo o op Mom @@ oO Oo 
Pod: 2 08 oo mo o 0 
° [ors oS ° S00 M00 00 
Co 9 of N wo 
re) 
2 3 4 5 1 2 x) 4 5 
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APPENDIX J. TWO FACTOR INTERACTION MODELS: HISTOGRAMS OF 
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APPENDIX K. TWO FACTOR INTERACTION MODELS 


VERSUS FITTED VALUES WITH LOESS SMOOTHING 
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APPENDIX L. TWO FACTOR INTERACTION MODELS: QQ-PLOTS 
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APPENDIX M. TWO FACTOR INTERACTION MODELS: ANALYSIS OF 


VARIANCE 
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factor (Community) 

factor (Squadron) 

factor (Rank) 

factor (Tot. Years) 

factor (Shop) 

factor (Shift) 

factor (Community) : factor (Rank) 
factor (Community) : factor (Tot. Years) 
factor (Community) : factor (Shop) 
factor (Community) : factor (Shift) 
factor (Squadron) : factor (Rank) 
factor (Squadron) : factor(Tot. Years) 
factor (Squadron) : factor (Shop) 
factor (Squadron) : factor (Shift) 
factor (Rank) :factor(Tot. Years) 
factor (Rank) : factor (Shop) 
factor (Rank) : factor (Shift) 
factor (Tot. Years) : factor (Shop) 
factor(Tot.Years):factor(Shift) 
factor(Shop): factor (Shifec) 
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factor (Community) : factor (Tot. Years) 
factor (Community) : factor (Shop) 
factor (Community) : factor (Shift) 
factor (Squadron) : factor (Rank) 
factor (Squadron) : factor(Tot.Years) 
factor (Squadron) : factor (Shop) 
factor (Squadron) : factor (Shift) 
factor (Rank) : factor(Tot. Years) 
factor (Rank) : factor (Shop) 
factor (Rank) : factor (Shift) 
factor (Tot. Years) : factor (Shop) 
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factor (Community) : factor (Rank) 
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factor (Community) : factor (Shop) 
factor(Community) : factor (Shift) 
factor (Squadron) : factor (Rank) 
factor (Squadron): factor (Tot. Years) 
factor (Squadron) : factor (Shop) 
factor (Squadron) ;-facrver (Shite) 
factor (Rank): factor(Tot. Years) 
factor (Rank) : factor (Shop) 
factor (Rank) : factor(Shift) 
factor(Tot. Years) : factor (Shop) 
Paccor( foe. ,ears) factor (Shift) 
factor (Shop) : factor (Shift) 
Residuals 
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F Value 
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factor (Squadron) : factor (Shop) 
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APPENDIX N. REDUCED SIMPLE MODELS 


Model: 
PA ~ factor (Squadron) + factor(Tot.Years) + factor(Shift) 


scale: 0.3056853 


DE Sum of sa RSS Cp 

<none> Pos20655 28125725 
factor(Squadron) 19 19.09854 284.1640 289.0550 
factor(Tot.Years) 6 6.98011 272.0456 284.8844 
factor (Shite) v2 2.98713 268.0526 283.9482 


Terms: 
factor(Squadron) factor(Tot.Years) factor(Shift) Residuals 
Sum of Squares 16.4369 722861 229387 11 265).0655 
Deg. of Freedom 19 6 AL 866 


Residual standard error: 0.5532452 
Estimated effects may be unbalanced 


QA ~ factor(Squadron) + factor(Tot.Years) + factor(Shift) 
scale: 0.4002939 


Df Sum of Sq RSS Cp 

<none> 343-6818 37052977 
factor(Squadron) 19 23.06513 371.7469 378.1516 
taceor (Tot.Years) 6  15.45358 364.1354) 380.9477 
factor (Shite) a O79004355849.5622 370.3975 


Terms: 
factor(Squadron) factor(Tot.Years) factor(Shift) Residuals 
Sum of Squares 22.4130 15.4952 0.9004 348.6818 
Deg. of Freedom 19 6 1 866 


Residual standard error: 0.6345352 
Estimated effects may be unbalanced 


Model: 
RS ~ factor(Squadron) + factor(Rank) + factor(Tot.Years) + factor(Shift) 


scale: 0.3476029 


DE Sum OL Sa RSS Cp 
<none> 301.9366 324.8784 
factor (Squadron) 19 16.56125 318.4979 328.2308 
factor(Rank) 6 B250006 307 24367 32622072 
factor(Tot.Years) 6 7.70716 309.6438 32624143 
Eactor (Shite). 2 it C495 56502-9861 9325-22327 
Terms: 
factor (Squadron) factor(Rank) factor(Tot.Years) factor(Shift) Residuals 
Sum of Squares 16.0094 18.0278 126563 1.0495 301.9366 
Deg. of Freedom 19 6 6 A 860 


Residual standard error: 0.5925277 
Estimated effects may be unbalanced 


Model: 
RM ~ factor(Squadron) + factor(Rank) + factor(Tot.Years) 


scale: 0.3237372 


DE Sum of Sq RSS Cp 
<none> 280.8129 301.5321 
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factor (Squadromjelo 23232159 300345 a2 so 
factor(Rank) 6 5.07801 285-0907 202 97250 
factor (Tot.Years) 6 10.54164 291,35426 308.1689 


Terms: 

factor (Squadron) factor(Rank) factor(Tot.Years) Residuals 
Sum of Squares 27.8628 19-2269 PO-5416 59260. 6129 
Deg. of Freedom 19 6 6 861 


Residual standard error: 0.5710932 
Estimated effects may be unbalanced 


Model: 
CC ~ factor(Squadron) + factor(Tot.Years) + factor(Shift) 


scale: 0.4031287 


Df Sum of Sq RSS Cp 

<none> 350.8909 372.6596 

factor (Squadron) 199227 16263) 373 2055593795046 
factor(Tot.Years) 6 17.43463 368.3255 385.2569 
factor (Shiecy Vet 2.80445 353.6953 374.6560 


Terms: 
factor (Squadron) factor(Tot.Years) factor(Shift) Residuals 
Sum of Squares 22209535 1 ie So 2.8045 350.8909 
Deg. of Freedom We) 6 i 866 


Residual standard error: 0.636542 
Estimated effects may be unbalanced 


Model: 
CR ~ factor(Community) + factor(Tot.Years) + factor(Shift) 


scale: 0.4686355 


Df Sum of Sq RSS cp 

<none> 418.7697 432.8288 

factor (Community) 7 —10.24208 429.0118 436.5100 
factor(Tot.vears) “6.21 -33¢50 440.1082 44675437 
factor (Shit) “1 1.89642 420.6661 433.7879 


Terms: 
factor(Community) factor(Tot.Years) factor(Shift) Residuals 
Sum of Squares 12.4423 Zee 63 1.8964 418.7697 
Deg. of Freedom yj 6 HE 878 


Residual standard error: 0.690622 
Estimated effects may be unbalanced 
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SCATTERPLOTS WITH 


APPENDIX O. REDUCED SIMPLE MODELS 


SUPERIMPOSED REGRESSION LINE 
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Fitted : tactor(Squadron) + factor(Tot. Years) + factor(Sht) 
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Fitted : factor(Community) + factor(Tot. Years) + factor(Shift} 
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APPENDIX P. REDUCED SIMPLE MODELS: HISTOGRAMS OF RESIDUALS 














o 
oO 
iv) 
S 
S Si 
2 i 
: ie | ra 
ran) ro) pect A | Eee 
-2 -1 0 1 -3 -2 “1 0 1 2 
resid(pa.step) resid(qa.step) 
i=) 
5 3 
oS 
S S 
S S 
© © 
2 “1 0 1 2 -3 2 -1 0 1 2 
CC resid(rs.step) CR resid(rm.step) 
oS 
oS oS 
isp) wo 
N 
S 
* % 
oS 
So 
ia ° 
w 
° ° 
-2 1 0 1 2 2 -1 0 1 2 
resid(cc.step) resid(cr.step) 


i: 


THIS PAGE INTENTIONALLY LEFT BLANK 


74 


RESIDUALS VERSUS FITTED 


APPENDIX Q. REDUCED SIMPLE MODELS 
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Fitted : factor{Squadron) + factor{Rank} + tactor(Tot. Years) 
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APPENDIX R. REDUCED SIMPLE MODELS: QQ-PLOTS 
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APPENDIX S. REDUCED TWO FACTOR INTERACTION MODELS 


Model: 
PA ~ factor(Community) + factor(Squadron) + factor(Tot.Years) + factor(Shift) + 
Boceor (Community) : factor (Shift) 


scale: 0.2751949 


DEsSsum,. of Sq RSS G9) 

<none> 259.7296 27735825 

factor (Squadron) leases. 259357 267.9649 279-5231 

factor (Tot.Years) 6 7.737269 267.4669 282.3274 

factor (Community) :factor(Shift) 5 5.319236 265.0488 280.4597 


Model: 
QA ~ factor(Community) + factor(Squadron) + factor(Tot.Years) + factor(Shift) + 
factor (Community) : factor (Shift) 


scale: 0.3641075 


DE Sum of Sq Rss Cp 

<none> 339.8904 363.925 

Eactlor (squadron). 12) 12.-36975 352. 26017367 35526 

factor (Tot.Years) 6 16.34776 356.2381 375.8999 

raccoon. Community) -Lactor(Shiftt) 5 8.42356 348.3139 368.7039 


Model: 
RS ~ factor(Squadron) + factor(Rank) + factor(Tot.Years) + factor(Shop) + factor(Shift) + 
factor( squadron) -factor (Shift) + factor(Tot.Years) :factor (Shift) 


scale: 0.3104461 


BE Sum.of sq RSS Cp 

<none> 23027776 51724703 

factor(Rank) 6 5.38959 236.1672 319-0745 

factor(Shop) 9 6.45093 23722286 318.2732 

baceor (Squadron) :faceom@surtt) 12 11.55724 292.3349 321.5168 
factor (Tot.Years):facter(Shift) 5 4.31919 295 09646-5139. 6250 


Model: 
RM ~ factor(Squadron) + factor(Rank) + factor(Tot.Years) + factor(Shop) + factor(Shift) + 
factor (Squadron) :factor(Shift) + factor(Tot.Years) :factor (Shift) 


scale: 0.2949249 


DE Sum of Sq Roo Cp 

<none> 299.6226) 290G2257 

factor(Rank) 6 3.99087 259.4134 290.6755 

factor (Shop) §.55127 260-9746 290.4663 

maetcor (Squadron) :factor(Shiéit) 12. 17.45745. 272.8600 300.5829 
factor (Tot .Years) :factor (Shift; 9 5 SS Oso re2 oe. 9991 290.eSle 


Model: 

cc ~ factor(Community) + factor(Squadron) + factor(Rank) + factor(Tot.Years) + 

factor (Shift) + factor(Commumity) :factor(Shift) + factor (Squadron) :factor(Snitejye+ 
factor (Rank) :factor(Smrre) + factor(Tot.Years) :factor (Shift) 


scale: 0.3595374 


D£ Sum of Sq RSS Cp 

<none> 324.3285 364.5967 

fiecomccomnunity) :factor(Shift) “0, “Oe0Q0000 324.3265 364-5967 
hRaGeonr (squadron) +factor (Shift) © 7 “5.543296 329.8716 365-1065 
factor (Rank) *factor (Snare) 65° “42227122 326255565505. 2285 
PaAeccouo (Tot. Years) :factor(Shift) S 5.275932 329-6045 366.2773 


We 


Model: 
CR ~ factor(Community) + factor(Rank) + factor(Tot.Years) + factor(Shift) + 


factor (Community); factor (Shift), + factor (Rank) -Eaceor(Shitt) + factor (tar. Years): 
factor (Shift) 


scale: 0.4254993 


D£ Sum of Sq RSS Cp 

<none> 393.4122 424.8992 

factor (Community) :- factor (Shitt) 96) 9-058145, 402 4703. 428. Ssi2 
factor (Rank):factor(Shift) 5 7.047982 400.4602 427.6921 
factor (Tot.Years) :Eactor(Shiit; “5 Je2e3000 400.6952 42878272 
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APPENDIX T. REDUCED TWO FACTOR INTERACTION MODELS: 
SCATTERPLOTS WITH SUPERIMPOSED REGRESSION 
LINE 
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APPENDIX U. REDUCED TWO FACTOR INTERACTION MODELS: 
HISTOGRAMS OF RESIDUALS 
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APPENDIX V. REDUCED TWO FACTOR INTERACTION MODELS: 
RESIDUALS VERSUS FITTED VALUES 
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APPENDIX W.REDUCED TWO FACTOR INTERACTION MODELS: QQ- 


PLOTS 
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APPENDIX X. COMPARISONS ON MODELS USING ANOVA 


The values listed in the following tables are the probabilities that the models are 
equivalent. Values printed in bold type indicate similar models. 

























Component Simple vs Simple vs. Simple vs. 
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APPENDIX Y. SUBJECT MATTER EXPERT RECOMMMENDED CHANGES 
TO MCAS DEMOGRAPHIC FACTORS 


Factors currently included in MCAS (no change): 


1. Community 
2. Squadron 

3. Rank 

4. Work Center 
> OTe 


Items to modify in MCAS: 
Total years of aviation maintenance experience (single item) 
modified to: 


6. Years worked in MOS 
7. Years worked outside MOS 


Factors to add to MCAS: 
education/training level 


8. attended an A school 
9. highest level of education attained 
10. number of maintenance activities assigned to during career 


morale/motivation indicators 


11. command advanced 

12. number of personal awards 

13. level of job satisfaction 

14. time to attain qualifications 

15. past performance on personal evaluations 
16. assigned to B tour 


other factors 


17. Age 

18. number of months in current squadron 

19, number of deployments 

20. supervisory designations earned 

21. level of confidence in Maintenance Control Officer 
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